System and method for facilitating generation and performance of on-line evaluations

ABSTRACT

A system and method for facilitating on-line generation and assignment of evaluations for entities of a particular organization.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. Ser. No.13/587,812, filed on Aug. 16, 2012, which is a continuation applicationof U.S. Ser. No. 12/981,757, filed on Dec. 30, 2010, which is acontinuation of U.S. Ser. No. 11/933,979, filed on Sep. 11, 2007, whichis a continuation of U.S. Ser. No. 10/106,933 filed on Mar. 25, 2002,which is based on and claims the benefit of the filing of U.S.Provisional Patent Application No. 60/278,299 filed Mar. 23, 2001, thecontents and disclosure of which are fully incorporated herein byreference.

BACKGROUND OF THE INVENTION

The Webster's English Dictionary defines evaluation as “to find thevalue or amount of or to judge the worth of.”

Websites currently on the market simply allow users to login andcomplete existing evaluations. In other words, these websites are a datacollection websites. Some examples include CollegeStats.com,RatingOnline.com and TeacherReviews.com.

For example: CollegeStats.com—anonymous online evaluation of teachers inany college or university. Teacher evaluations are then posted on thewebsite for any other students to see. The site is “run by students, forstudents, and is completely free”. RatingOnline.com—an online resourceto find out “what the other students think of the professors at yourcollege”. The service allows students to find out whether the professorassigns and grades homework, number of quizzes and exams, and if theprofessor gives extra credit. StudentInfo.org—student-based websitefocusing on California colleges and universities, providing professorreviews to interested students. TeacherReview.com—student-run websitecollecting information from students about colleges across the globe,students complete evaluations to help other students select classes.Aceil.com—dedicated to the advancement of international academicexchange by facilitating the “evaluation of all international academicand professional credentials”.

EducationEvaluation.com—provides online applications for variousindividuals to “clarify and verify credentials, degrees, and diplomas”.Evaluations.com—online data collection of customized questions submittedin writing. Participants may include any person fitting the marketingrequirements of a client company. Client companies submit the request inwriting and Evaluations.com generates marketing questions based on thoserequirements. Primary business is to provide online marketing research.AgentFinders.com—site reports the performance Atlanta real estateagents. Service researches real estate market activity for each clientrequest and provides data analysis in faxed or mailed reports.

There is one company on the market (Evaluations.com) that allows usersto contact the company and design questions that are then implementedinto an on-line evaluation. However, they do not support the ability tobuild evaluations online. Currently, there is no web-based evaluationsystem that allows users to build their own evaluations.

Further, many organizations that provide web-based evaluations allowregistered users to be assigned to a specific evaluation or marketingquestionnaire. However, these are tasks that are primarily completedinternally by the organization or manually by a list. Alternatively,some web-based evaluation systems allow participants to be includedbased on the specific marketing requirements of a client company. Noneof the existing companies allow the administrative staff to make theirown assignments.

Further, there are no organizations that allow administrative users of aweb-based evaluation system to generate real-time online reports basedon the data collected.

It is desirable to provide a comprehensive on-line facilitator thatprovides registered users all the tools necessary to build evaluationquestions, individual evaluations, and to assign these evaluations toregistered users within their organization.

It is further desirable to provide a comprehensive on-line web-basedevaluation review system that enables collection of evaluation data and,once collected, enables registered users to access their account andreview the results of these evaluations. More specifically, in the areaof medical education, the facilitation of periodic scheduled evaluationbetween students and physicians for the enhancement of medical trainingand satisfaction of Board and regulatory agency requirements.

BRIEF SUMMARY OF THE INVENTION

Referred to herein as MYEVALUATIONS.COM®, there is provided a web-basedsystem that allows a user to literally design a complete evaluationsystem online, by providing a framework with pre-written and customizedquestions, and answer choices. In addition to providing on-lineevaluations for customers, the MYEVALUATIONS.COM® web-based systemprovides other distinct functionality that sets it apart from all otherweb-based evaluation systems. These include:

-   -   1) the ability to custom build Evaluations^(SM) completely        online;    -   2) the ability to customize evaluations to an individual        organization;    -   3) the ability to implement a comprehensive database of        evaluation questions;    -   4) the ability to add custom questions;    -   5) the ability to assign Evaluations^(SM), e.g., evaluations may        be assigned to medical doctors, residents, nurses and fellows in        order to provide 360 degree field of evaluations within the        hospital environment;    -   6) the ability to assign evaluations to a group or to an        individual;    -   7) the ability to make assignments according to an individual's        name, sub-specialty, or post-graduate year;    -   8) the ability to assign evaluations by month, quarter, year, or        a specific date or date range;    -   9) the ability to monitor pending evaluations and compliance;    -   10) the ability to automatically e-mail notification of        assignments to reviewers;    -   11) the ability to automatically e-mail reminders of incomplete        evaluations; and,    -   12) the ability for users to log on and complete ad-hoc        evaluations.

Specifically, system functionality additionally includes the ability toreview generated Evaluations^(SM) including the ability to generateclear, concise, and comprehensive online reports from the datacollected; and, the ability to track medical doctor, resident, nurse andfellow completion of evaluations. Thus, besides enabling building ofevaluations online, a unique feature of MYEVALUATIONS.COM® is theability to enable any administrative user to make assignments. Forexample, in the case of hospitals, the services provided byMYEVALUATIONS.COM® web-site enable the administrative staff to setup theusers within a hospital. The second step is to design the onlineevaluations. The third step is to assign specific evaluations.

Assignments are made online and can be targeted to any registered memberof the hospital. The assignments can be made by an individual's name,sub-specialty, or post-graduate year. These assignments can be made bymonth, by quarter, by year, or for a specific date. In addition, userscan log on and complete ad-hoc evaluations. Individuals are thenautomatically notified by e-mail regarding pending and/or overdueevaluations. Therefore, the hospital has a streamline system that allowsthem to completely manage their internal evaluation system.

As a fourth step, MYEVALUATIONS.COM® services enable the generation ofonline reports based on the data collected. A unique feature ofMYEVALUATIONS.COM is the ability to generate a real-time and onlinereport based on the data collected via the online evaluations. Thecontent of these reports is absolutely unique in design, by generatingspecific information on average performance and percentile ranking forindividuals or a group, and cross-comparing the data between individualsor groups.

Advantageously, the present invention targeted and designed for academichospitals with users that include hospital administration, staff,medical doctors (i.e., Attendings), nurses, residents and fellows.Individual hospitals are able to register for services provided by theMYEVALUATIONS.COM® web-site by paying a monthly fee. Once a hospital isregistered the administrative staff can set up the staff, medicaldoctors, nurses, residents, medical students and fellows that make upthe actual evaluation users. Once users have been set up, theadministrative staff can design completely customizable evaluations thatcan then be assigned to the actual evaluation users. After assignmentsare made and users have answered the evaluation questions, theadministrative staff can generate group-specific reports for internalreview.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The above and further advantages of the present invention may be betterunderstood by referring to the following description in conjunction withthe accompanying drawings, in which:

FIG. 1 is a diagram illustrating an Internet/Web-based communicationssystem 10 established for enabling on-line evaluationsbuild/assignment/management and reporting functionality and procedurebuild/tracking functionality according to the invention;

FIG. 2 illustrates an example MYEVALUATIONS.COM® log-in web page 50;

FIG. 3( a) illustrates an example Main menu comprising the defaultinterface display 60 presented when a user logs on, and FIG. 3( b)illustrates an example Main menu comprising a default interface display70 presented when an administrator user logs on;

FIG. 4 illustrates an example screen 80 at the Administrator AccessLevel tor enabling the management of user profiles, management ofevaluations, management of hospital/department profiles, reportgenerations, et al;

FIG. 5( a) illustrates an exemplary Add User Profiles interface 85 andFIG. 5( b) illustrates an exemplary user profile data entry interfacefor a Resident user to enable the administrator to enter all relevantdata pertaining to that new individual user;

FIG. 6 illustrates an Edit Hospital/Department Profile interface 100 forenabling the editing/updating of hospital/department profiles formedical profession evaluations;

FIG. 7 illustrates an example interface 110 which initiates presentationof a build evaluations Evaluations Build option of FIGS. 8( a)-8(d).

FIGS. 8( a)-8(d) provide various interfaces for initiating processes tobuild/design and edit an evaluation;

FIG. 9( a) illustrates an example Evaluation Assignments menu interface200 and FIG. 9( b) illustrates example types of assignments availablevia the system;

FIG. 10 illustrates an example Evaluation Assignment interface 210downloaded to users for assigning an evaluation to residents, forexample, who will be evaluating attending(s), and vice versa.

FIG. 11 illustrates an example Manage Evaluation Assignments interface250 for enabling the management of previously assigned evaluations;

FIG. 12 illustrates an example Manage Evaluations Questions interface260 for enabling the maintenance and addition of questions pertaining tocompetencies that may be selected to comprise an evaluation;

FIG. 13 illustrates a web-based interface display 270 providingfunctionality for adding an evaluation question to the evaluationquestions database 18 a of FIG. 1;

FIG. 14 illustrates an example Reports Selection interface 300 forinitiating real-time, on-line generation of various management reports;

FIG. 15( a) illustrates an example Individual Resident Evaluationsreport interface 314 providing functionality for on-line generation ofan Individual Resident Evaluations report 312 an example of which isdepicted in FIG. 15( b);

FIG. 16( a) illustrates an example Summary of Group/Resident Evaluationsreport interface 323 providing functionality for on-line generation of aSummary of Group/Resident Evaluations report 321 an example of which isdepicted in FIG. 16( b);

FIG. 17( a) illustrates an example Summary Peer-to-Peer Evaluationsreport interface 333 providing functionality for on-line generation of aSummary Peer-to-Peer Evaluations report 331 an example of which isdepicted in FIG. 17( b);

FIG. 18( a) illustrates an example Summary Program Evaluations repostinterface 343 providing functionality for on-line generation of aSummary Program Evaluations report 341 an example of which is depictedin FIG. 18( b);

FIG. 19( a) illustrates an example Trending of Resident Performancereport interface 353 providing functionality for on-line generation of aTrending of Resident Performance report 351 an example of which isdepicted in FIG. 19( b);

FIG. 20 illustrates an example Summary of Resident's Core Competenciesevaluation report interface 363 used for generating a Resident's CoreCompetencies evaluation summary report;

FIG. 21( a) illustrates an example the Overdue Evaluations report 372;FIG. 21( b) illustrates an example e-mail communication 59 sent to anevaluator providing the evaluation assignments due/overdue;

FIG. 22( a) illustrates an example Resident Completion Status reportinterface 383 providing functionality for on-line generation of aResident Completion Status report 381 an example of which is depicted inFIG. 22( b);

FIG. 23( a) illustrates an example Class Rank report interface 393providing functionality for on-line generation of a Class Rank report391 an example of which is depicted in FIG. 23( b);

FIG. 24( a) illustrates an example Early Warning report interface 403providing functionality for on-line generation of an Early Warningreport 401 an example of which is depicted in FIG. 24( b);

FIG. 25( a) illustrates an example Resident Comments report interface413 providing functionality for on-line generation of a ResidentComments report 411 an example of which is depicted in FIG. 25( b);

FIG. 26 illustrates an example Procedures Menu Interface 500 forenabling the generation, management and tracking of procedures;

FIG. 27 illustrates an example interface 510 for enabling auser/administrator to add a new completed procedure on-line;

FIG. 28 illustrates an example web-based communication providing aninterface 530 for modifying submitted procedures;

FIG. 29 illustrates an example web-based communication providing aninterface 550 for generating a detailed list of all procedures andfunctionality for initiating addition of new procedures;

FIG. 30 illustrates an example web-based communication providing aninterface 560 for designing a new procedure; and,

FIG. 31 illustrates the example web-based communication providing aninterface 570 for selecting an attending who is to supervise theprocedure.

DETAILED DESCRIPTION OF THE INVENTION

MYEVALUATIONS.COM® is a web-based service that provides registered userswith all of the tools necessary to build evaluation questions,individual evaluations, and to assign these evaluations to registeredusers within different organizations, on-line. Once the data iscollected, registered users can access their account and review theresults of these evaluations. As will be described, the evaluationsservices in the preferred embodiment are targeted and designed foracademic hospitals. The users include hospital administration, staff,medical doctors, nurses, residents, medical students and fellows.Individual hospitals are able to register for the evaluations servicesby paying a monthly fee plus a one-time setup fee. Once a hospital isregistered, the administrative staff can set up the staff, medicaldoctors, nurses, residents, medical students and fellows that make upthe actual evaluation users. Once users have been set up, theadministrative staff may design completely customizable evaluations thatcan then be assigned to the actual evaluation users. After assignmentsare made and users have answered the evaluation questions, theadministrative staff may generate group-specific reports for internalreview.

FIG. 1 is a diagram illustrating an Internet/Web-based communicationssystem 10 established for enabling on-line evaluationsbuild/assignment/management and reporting functionality according to theinvention. As shown in FIG. 1, the invention comprises a web site 31,maintained and operated by MYEVALUATIONS.COM®, providing the secureon-line connection service over the Internet, that includes one or moreweb/database servers 30 comprising application and database softwarecomponents for building and assigning evaluations online and reportingevaluation data. Registered users 12 a, . . . , 12 n of the web site areenabled to access the web site 31 remotely via wired or wirelessconnections to the Web/Internet 15. Wired communications between the website 31 and the registered users are via the public Internet inaccordance with standard TCP/IP protocols and optionally, over a securecommunications link, e.g., secure sockets layer, BlueTooth or similarprotocol. It is understood that parties 12 a, . . . , 12 n may accessthe Web/Internet via a personal computer/computing device, persondigital assistant, or like device implementing web-browserfunctionality, e.g., Netscape® or Internet Explorer®, or other browsingtechnology that may be compatible.

The MYEVALUATIONS.COM® web-site includes one or more web-servers 30executing a collection of web-based applications implementing, forexample, Active Server Page (ASP), JavaScript, HTML, VB Script with aSQL Server database. This preferably operates on a centralized server 30and database with 128-bit security. Provided at a website server 30 arevarious Internet Information Services (IIS) which are mechanismsenabling files on a computer to be read by remote computers andparticularly, used to house, secure and present a web site to either theInternet or an intranet (private network); and Component Services (COM)which function as a repository of custom Dynamic Link Libraries (dll's)that allow custom applications to perform actions in data sourcesforeign to the application, e.g., enabling a web page to query data on adatabase.

As shown so FIG. 1, a centralized database may be partitioned intoseveral databases including a hospital/department profiles database 39 afor storing respective profiles of the registered clients, e.g.,hospitals and their various departments represented by a computerworkstation at a hospital 21, and a users profiles database 39 bassociated with the hospital/department database for storing informationfor the various registered users (e.g., evaluators/evaluatees 12 a, . .. , 12 n) or administrator 20. The system 10 further includes aquestions and answers database 18 a, 18 b, respectively, for storingquestions and various answer choices to be used in the evaluationsgenerated; an evaluations database 18 c for storing the builtevaluations for on-line use; an assignments database 29 for storingassignment information pertaining to the evaluators (assignees) of theon-line evaluations; and, a procedures database 34 for storing builtprocedures required to be performed by residents and medical students,for example, and used for tracking purposes as will be described ingreater detail herein. As will be described in greater detail herein,the web-server 30 preferably executes a variety of application specificprograms, including, but not limited to: an evaluations build softwaremodule 24 providing functionality for building evaluations to beassigned and stored in the evaluations database 18 c; an evaluationassignment software module 26 providing functionality for assigningevaluations; an overdue evaluation scan module 28 providingfunctionality for tracking status of pending assigned evaluations; areport generator module providing functionality for generating a varietyof reports; and a procedure build/tracking module 25 providingfunctionality for generating procedures and tracking the performance ofprocedures in satisfaction of Board and/or regulatory agencyrequirements, in the manner as will be explained in greater detail.

Login

FIG. 2 illustrates an example MYEVALUATIONS.COM® log-in web-basedcommunication (e.g., web-page 50) including a username field 51 andpassword field 53. Users may access the system using a pre-assignedusername and password. If a registered user has forgotten their passwordor user name, user may retrieve this information via e-mail using awell-known forgotten user password function 52. The username may beautomatically generated based on the user's first name and last name.When a user is added to the users database 39 b (FIG. 1), the systemwill use the first initial from the first name and combine it with thelast name in order to generate a complete username (e.g., John Adamswill be JAdams). The application may then scan the database to assurethat the username is unique. If the selected username already exists inthe database, then a number (starting with 1 up to ∞) will be added tothe end of the username (e.g., JAdams1). The username can be optionallycase sensitive; otherwise by default it will not be case sensitive. Thepassword field 53 may require a minimum of four characters. Thecharacters can be a combination of letters, numbers, and non-restrictedsymbols. The first time a user logs into MYEVALUATIONS.COM® he(s)he willbe prompted to change their user to a new confidential password. Thepassword can be optionally case sensitive; otherwise by default it willnot be case sensitive.

User Access

By entering their unique username and password each user will haveaccess to their evaluations and records. There are two general useraccess levels: an Administrator level which includes all individualsneeding system access to administrative features such as Manage UserProfiles, Design Evaluations, and Reports (e.g., FIG. 3( b)); and, aUser level which includes all individuals needing system access forcompleting evaluations and personal reports (e.g., FIG. 3( a)).

Each general user who logs on will have access to the following MainMenu options as shown in the example downloaded web-based communication60 of FIG. 3( a) providing functionality for a particular registereduser (e.g., hospital department 61) of the system 10. More particularly,from the main (Default) screen 60, the user may select the followingoptions: a Voluntary option 62 providing user access to voluntary orAd-hoc evaluations; a Procedures option 64 providing users access toon-line procedure submission and tracking; a Reports option 65 providingaccess to personal reports; a Password option 63 providing user accessto change personal profile including password; and a Logoff option 66enabling users to log-off user from the web-site.

It should be understood that each user registered as a systemadministrator or user/administrator (of a hospital department 61) whologs on the system will have access to the following Main Menu optionsas shown in the example downloaded web-based communication 70 of FIG. 3(b) providing additional functionality including: the Main (Default)screen option 70 as seen when a user logs on; a Mail option 72 whichprovides access to user's e-mail for sending individual or groupe-mails; a Reports option 65 providing user access to personal reports;an Evaluations option 75 enabling the administrator system access todesign, assign and manage department evaluations; the Procedures options64 providing the administrator with access to additional on-lineprocedure tracking and management functionality; the Users option 76providing access to manage user profiles; a Setup option 78 providingaccess to modify the department's profile and academic calendar; thePassword option 63 providing access to change personal profile includinguser's password; and, the Logoff option 66. For users having a combinedUser/Administrator profile, they will have access to the Main, Mail,Voluntary, Reports, Evaluations, Procedures, Users, Setup, Password, andLogoff options.

From the main menu interface 70 of FIG. 3( b), an administrator may addnew users by selecting the Users option 76 which initiates presentationof a Manage User Profiles interface 80 such as shown in FIG. 4. It isfrom this interface that enables Administrator access to enable themanagement of user profiles, management of evaluations, management ofhospital/department profiles, report generations, etc. From this screen,links are provided that enable functionality for system administratorswho add user/hospital department profiles and manage base evaluationquestions. As shown in FIG. 4, the Manage User Profiles interface 80included an Add User Profile button 82 which, when selected, generatesfor download to the administrator an Add User Profiles interface 85 suchas shown in FIG. 5( a). The downloaded Add User Profiles interface 85provides functionality for selecting the type of user (e.g., Resident,Chief Resident, Attending, Fellow, Nurse, Medical Student or Staff) byselecting an appropriate radio button 86; and, pressing a “next” button87 to advance to the next page, or “Cancel” to stop adding a new user.In response to pressing the “Next” button 87, a new web-based interfaceis communicated such as the example Add Resident User Profile interface90 shown in FIG. 5( b) to enable the administrator to enter all relevantdata pertaining to that new individual user in entry fields forentering(s)electing new user information including: the user's firstname 91; the user's last name 92; the user's password 93; that user'saccess level 94 where the default value is “User” and should be used forall individuals needing access for completing evaluations; or,alternately, the access level is “Administrator” and should be used forall individuals needing access to administrative features such as ManageUser Profiles, Design Evaluations, Procedures and Reports, etc; a YearFirst Started value 96 which is used to calculate a Post-Graduate Year(“PGY”) and/or total number of years employed; optionally entering thatuser's pager number 97 which data may eventually be printed on an“Overdue Evaluations” report, and, on the user list page, as will beexplained in greater detail herein; optionally entering that user'stelephone number 98; that user's e-mail address 99 which data is used tosend automatic notifications of pending and overdue evaluations. This isalso used as the e-mail address for forwarding a forgotten password;and, that user's Default E-mail Address (not shown) which is an optionthat is only available under the Chief Resident user type and is usedfor all users who do not have an e-mail address on file. It should beunderstood that the Username is automatically generated based on theuser's first initial and last name. If the user name is already in use,then a number (starting from 1) will be placed at the end of theusername.

Further with respect to managing user profiles, the downloaded web-basedcommunication such as the example web-page illustrated in FIG. 4,displays the names of all current users. That is, FIG. 4 illustrates anexample user profile management screen 80 for enabling the management(sorts and orders) of users. As shown in FIG. 4, all of the currentusers are displayed in a formatted table 81; the table including aheading row. Selecting the heading row will cause the entire table tosort based on the contents in the selected column. The table willinclude the following headings: 1) Edit 83: This function will permitthe editing of individual user profiles. Selecting this functionlaunches the user edit page similar to the add user profile page; 2)Delete 84: This function will permit the deletion of the individual userprofile. It is important to note that this action will only causedeletion of the profile and not the associated evaluation data.Selecting this function additionally launches a confirmationcommunication (e.g. web-page) to confirm deletion by pressing “OK” or to“Cancel” deletion of the selected user profile; 3) User ID 88: The datafrom Username will be displayed in this column and is automaticallygenerated based on the user's first initial and last name. If the username is already in use, then a number (starting from 1) will be placedat the end of the username; 4) Password 93: the password of a registereduser; 5) First Name 91: The first name of a registered user; 6) LastName 92: The last name of a registered user; 7) User Type 94: causesdisplay of data from User Type, including Resident, Chief Resident,Attending, Fellow, Nurse, Medical Student or Staff; 8) a First Year 96:display data from Year First Started which represents the year theindividual first started working at the selected Department; 9) E-mailAddress 99: the e-mail of the registered user; and, the pager of theregistered user (not shown).

The Manage User Profiles function 80 of FIG. 4 may be sorted andmanipulated in many ways. As outlined above, each heading column may beused to sort the table 81. In addition the top of the table will includethe following functions: 1) Filter User Type 89 a providing means tosort users by user type. Selecting a category from this list willdisplay only individuals that are within the specified category. Todisplay all individuals select the (All) option from the drop-down menuand then select: a Resident/Chief Resident; Attending; Fellow; Nurse;Medical Student; Staff, for example; and, 2) Filter Column and Value 89b, 89 c providing means to sort users by user information by enablingselection an option from the Filter Column. Selecting a category fromthis list will display only individuals that are within the specifiedcategory. To display all individuals, an administrator may select the‘Refresh’ button 89 d from the screen 80. The options include: LastName, Firstname Name, User ID, (first initial and last name), and, FirstYear (year first started). Next, the user may type a Filter Value 89 cto find a user(s) based on specific criteria. This field is used incombination with the “Filter Column” option, (e.g., Select “FilterColumn” option “First Name” and type the first few letters of a person'slast name and press Refresh.) and the query results will be displayed inthe table 81.

From the main menu interface 70 of FIG. 3( b), an administrator maysetup a department profile by selecting the Setup option 78 whichinitiates presentation of a Edit Hospital/Department Profile interface100 such as shown in FIG. 6. From this interface, an administrator maysetup the department profile by entering information including: ahospital name 102, a program type 104, and address/contact information105. Further entries include selection of the first day of the academicyear in month day 107 which is used to set the commencement date of the12 or 13 month calendar year, and the maximum number of evaluationexemptions 109 which are assigned to users in some circumstances toprovide users with an option not to complete an assigned evaluation. Itis understood that each time an evaluation exemption is exercised by auser, that user's exemption number total is decremented automatically.

In order to streamline the assignment functions, an automated calendarfunction is implemented. Particularly, the administrator for thedepartment may select two options from the Setup Menu in order todetermine their academic schedule and rotation blocks: 1) Number ofRotation Block (RB) per academic year which is either 12 or 13 block.That is, based on a 12-RB schedule the system will automaticallycalculate the start and end of each RB to correspond to the start andlast day of each calendar month. The starting month of the 12-RB yearwill be based on the First Day of the Academic Year 107, or, based on a13-RB schedule the system will automatically calculate the start and endof each RB based on a 28-day cycle independent of the calendar month.The starting day of the 13-RB year will be based on the First Day of theAcademic Year 107; 2) a First Day of the Academic Year. The Departmentwill select the month and date of the first day of the current academicyear. This will represent the first working day for either the 12 or13-RB schedule. From a Program Year Calendar option 106, an on-dinecalendar representing the months of the current academic year ispresented to the user (not shown). By specifying the required data inthe Calendar Function 107, the entire academic year's schedule can beseen at a glance. The start of each Rotation Block is highlighted inyellow and represents either a monthly or 28-day interval from the startof the academic date, depending upon whether the 12-RB or 13-RB year isimplemented.

In the preferred embodiment, the setup provides an Auto NotificationFunction in order to notify users of overdue and incomplete evaluations,such as the tables 55 provided in the default main menu screens 60,70 ofFIGS. 3( a) and 3(b), respectively. This notification function willautomatically notify users of overdue evaluations and the department hasthe option 108 to turn-off this function from the Setup Menu 100. TheAuto Notification Function 108 particularly will periodically (e.g.,daily) initiate automatic scanning of the assignments database 29(FIG. 1) for all pending and overdue evaluations and automatically senden e-mail reminder to each respective user. In an example embodiment,the notifications are sent for every seven days they are past due. Forexample, if an assignment was due Jan. 1, 2001 and remains pending, thesystem will send an e-mail notification on Jan. 8, 2001, Jan. 15, 2001,and Jan. 22, 2001 and so forth.

Thus, it should be understood that Evaluators know they have evaluationsdue in one of two ways: 1) by logging on to MYEVALUATIONS.COM® whichautomatically displays their evaluation assignments in an “Evaluationsto be Complete” table 55 as illustrated in the Main menu screens 60, 70provided in the example user main menu screen of FIGS. 3( a), 3(b); or,2) as will be described, they receive an e-mail with assignmentsdue/overdue such as shown in example e-mail notification 59 provided inFIG. 21 (b).

Design/Build Evaluation

From the main menu interface 70 of FIG. 3( b), a use/administrator maybuild an evaluation by selecting the Evaluations option 75 whichinitiates presentation of a build evaluations interface 110 such asshown in the example interface of FIG. 7. From this interface 110, theuser is provided with selection options to design an evaluation 112 thatis customizable to individual organization; edit an evaluation 114;edit/manage the database of on-line evaluations 118 including theability to implement a comprehensive database of pre-written evaluationquestions, ability to add custom questions, ability to choose answerchoices, etc; and the ability to assign evaluations 116.

The selection of the design an evaluation 112 option initiates thegeneration of a web-based interface 120 such as shown in FIG. 8( a) thatprovides the first of a four (4) step process for building an on-lineevaluation. The fully automated feature guide the user step-by-step fordesigning a customized on-line evaluation.

With respect to the example evaluations build interface 120 of FIG. 8(a), which implements the evaluations build module 24 of FIG. 1, processstep are implemented for selecting the target audience 123; selectingthe subject audience 125; specifying an evaluation title 126; specifyingthe requirements for comments 127; and, specifying whether theevaluation may be used as a voluntary evaluation, i.e., evaluationavailability 129. The process continues by selecting a ‘Next’ button toadvance to the next step, or ‘Cancel’ to exit and return to theEvaluations menu 110.

With respect to step 1, the step of selecting an audience is to specifywho will be the evaluator and may include, for example: residents,medical students, fellows, nurses or attendings. If a resident is anevaluator, he(s)he will answer questions about the evaluatee, who may bemedical students, fellows, nurses or attendings, or all. Theuser/administrator selects the Resident target audience from the dropdown menu to design an evaluation targeted to the resident audience. Ifthe medical student is an evaluator, he(s)he will answer questions aboutthe evaluatee, who may be residents, fellows, nurses or attendings, orall. The user/administrator selects the student target audience from thedrop down menu to design an evaluation targeted to the residentaudience. If a fellow is an evaluator, he(s)he will answer questionsabout the evaluatee, who may be resident, medical students, nurses orattendings, or all. The user/administrator selects the fellow targetaudience from the drop down menu to design an evaluation targeted to theresident audience. If a nurse is an evaluator, he(s)he will answerquestions about the evaluatee, who may be residents, medical students,fellows or attendings, or all. The user/administrator selects the nursetarget audience from the drop down menu to design an evaluation targetedto the resident audience. If an attending is an evaluator, he(s)he willanswer questions about the evaluatee, who may be residents, medicalstudents, fellows, or nurses, or all. The user/administrator selects theattending target audience from the drop down menu to design anevaluation targeted to the resident audience. The other option in thisstep is to select a unique title 126 for the evaluation that is beingdeigned, e.g. “End-of-Rotation Evaluation” or “Resident MonthlyEvaluation.” Preferably, a generic title is used in order to use thesame evaluation year-to-year so that data may then be used for trendingand comparison reports. For example, unique titles may be used in orderto differentiate evaluations that may be assigned in series. Forexample: “2001-2002 Monthly Resident Evaluation.” Additionally, it ispreferred that a unique title for evaluations and questions that changefrom period-to-period be used.

With respect to specification of whether comments are required 127, thisis to inform whether the evaluator is required to write comments inorder to complete an evaluation. Each evaluation will have a comment boxfor the evaluator to write comments and the option to make commentsmandatory is YES (default) or NO (optional). If Yes is specified, theevaluator will be required to enter comments when completing anevaluation. If the evaluator forgets to enter comments, they will beautomatically prompted to write comments. If No is specified, theevaluator will not be required to enter comments. A comment box will beprovided for optional comments.

With respect to whether the evaluation being designed may be used forvoluntary evaluations 129, the designer may choose to make evaluationsavailable for voluntary submission. When a resident or attending logsonto MYEVALUATIONS.COM® he(s)he is presented with a list of assignedevaluations. In addition to assigning specific evaluations to residentsand attendings (mandatory evaluations), evaluators have the option tocomplete voluntary evaluations. This is a valuable tool for evaluatingspecific individuals that were not assigned to an evaluator. Forexample, a Resident rotating through Infectious Diseases is assigned amandatory end-of-rotation evaluation on each of his three ID attendings(mandatory evaluations). During the course of the month she spends agreat deal of time rounding with the Chief of Pharmacy and now wants toevaluate him. Since there are no mandatory evaluation assigned to her tocomplete on the Chief of Pharmacy, she may choose to complete avoluntary evaluation. By choosing to complete a voluntary evaluation,the evaluator may select the Evaluation and Evaluator. Only evaluationsthat are marked as voluntary-YES (default) will be made available toevaluators completing voluntary evaluations. The administrator canchange the status of an evaluation from Voluntary to Non-Voluntary atanytime (see section Edit Evaluations). If YES is selected, then theevaluation is to be included as part of the list of evaluationsavailable to evaluators wanting to complete voluntary evaluations; ifNo, the evaluation will not be included as part of the list ofevaluations available to evaluators wanting to complete voluntaryevaluations. However the evaluation will be available for mandatoryevaluations, when making assignments, as will be explained in greaterdetail herein.

As mentioned, for each evaluation created there is associated a set ofquestions and answer choice responses as shown in FIG. 1 databases 18a,b. The second step of designing an evaluation involves the step ofselecting the answer choice types, which comprise a scale in onepreferred embodiment, and selecting a respective category as illustratedin the example received web-based communications 130 of FIGS. 8(b)(1)-8(b)(5). From each of these displays, selecting ‘Next’ willadvance to the next (third) design step, and ‘Back’ to return to stepone, or ‘Cancel’ to exit and return to the Evaluations menu.

As shown in FIGS. 8( b)(1)-8(b)(5), with respect to selection of answerchoice response types, interface 130 provides a drop-down menu fromwhich a user may select the answer scale granularity. There are fivemajor answer scales that may be selected from this menu including: aScale 134 of 0-5 (0=lowest and 5=Highest) as shown in FIG. 8( b)(1); aScale 140 of 0-10 (0=lowest and 10=highest) as shown in FIG. 8( b)(2); aScale 150 of 0-9 (0=lowest and 9=highest) as shown in FIG. 8( b)(3); aScale 160 of A-F (F=lowest and A+=Highest) as shown in FIG. 8( b)(4);and, a Scale of Yes/No (Yes=9 and No=1) as shown in FIG. 8( b)(5). Auser may choose the respective category by pressing the button next tothe category name.

With respect to selection of the Scale 134 of 0-5 in FIG. 8( b)(1),there are six answer choice categories under this scale. These representthe actual answer choices an evaluator will see when completing anevaluation. A user will only have one answer choice type per evaluation.The user/administrator chooses the desired category by pressing thebutton: next to the category name. Categories in the 0-5 scale include:a Numbers Only category 135, i.e., numbers without descriptors; aPartial Category 136, i.e., numbers with descriptions for numbers 1, 3and 5; a Traditional Answer Category 131, i.e., numbers with traditionaldescriptors; a Conservative Category 136, i.e., numbers withconservative descriptors; an Answer Category 139, i.e., numbers withconformity descriptors; and, a Casual Category 132, i.e., numbers withcasual descriptors.

With respect to selection of the Scale 140 of 0-10 in FIG. 8( b)(2),there are three answer choice categories under this scale that are theactual answer choices an evaluator will see when completing anevaluation. A user can only have one answer choice type per evaluation.The user/administrator chooses the desired category by pressing thebutton next to the category name. Categories in the 0-10 scale include:the Numbers Only category (without descriptors) 142; a Partial Category144, e.g., having numbers with descriptions for numbers 1, 5 and 10; anda Traditional Answer Category 146, e.g., numbers with traditionaldescriptions for numbers 1, 5 and 10.

With respect to selection of the Scale 150 of 0-9 in FIG. 8( b)(3),there are eight answer choice categories under this scale, including the9-point Sliding Scale answer choices for the two-part questions (as usedby the American Board of Internal Medicine (ABIM)). Preferably, thereare nine answer choice categories under this scale, including the9-point Sliding Scale answer choices for the 2, 3, and 4-partedquestions (as used by the ABIM). These are the actual answer choices anevaluator will see when completing an evaluation. A user will only haveone answer choice type per evaluation by choosing the desired categoryby pressing the button next to the category name. The principaladvantage of the nine-point scale over shorter scales are the additionallevels of discrimination it provides. Thus, scales that include morerating steps tend to produce more reliable ratings. When using the9-point rating scale, a rating of 4 is defined as “marginal” and shouldconvey a message that remediation is necessary. A clearly satisfactoryresident should receive a rating of “5”. Categories in the 0-9 scale 150include: a Numbers Only 151, i.e., numbers without descriptors; aPartial Category Standard 154, i.e., numbers with descriptions fornumbers 1,4, 5 and 7; a Partial Category Modified 157, i.e., numberswith descriptions for numbers 1, 5 and 9; a Partial Category Shifted152, i.e., numbers with descriptions for numbers 2, 5 and 8; aPartial-Compete Categories 155, i.e., numbers with descriptions fornumbers 2, 4, 5, 6 and 8; a Complete Categories 158, i.e., numbers withdescriptions for all numbers; Categories Only 153, i.e., descriptionsonly without numbers; a Rating Categories 156, i.e., descriptions withratings so that 5 conveys “Expected level of performance,” rather than“Satisfactory”; and, Sliding Scale 159, i.e., numbers with descriptionsfor numbers 2, 4, 5 and 8. Select the Sliding Scale answer category inorder to design a two-tiered evaluation (as used on the ABIM ResidentEvaluations). This answer choice is only available when designing anevaluation for the Attending audience.

With respect to selection of A-F scale 160 in FIG. 8( b)(4), there arethree answer choice categories under this scale that are the actualanswer choices an evaluator will see when completing an evaluation.Categories in the A-F scale 160 include: a Letters Only category 162,e.g., numbers with letters, a partial Categories 164 with numbers andletters with descriptions for numbers, e.g., 2/D, 5/B and 9/A+; and,Complete Categories 166, e.g., numbers and letters with traditionaldescriptions for all numbers The remaining Yes/No scale 170 in FIG. 8(b)(5) includes a Yes/No category 175, for example.

The third step of designing an evaluation involves the step of selectingthe question to include in the evaluation for storage in questionsdatabase 18(a) in FIG. 1. This step is illustrated in the exampleweb-based communication 180 of FIG. 8( c). As shown in FIG. 8( c), auser may preferably scroll through a list of questions 182 and mark eachquestion to include in the evaluation with a check mark in box 187; thenselect ‘Next’ to advance to the next step, or ‘Back’ to return to Steptwo, or ‘Cancel’ to exit and return to the Evaluations menu. Questionsare categorized into competencies ______, which are sortedalphabetically. More particularly the user will be presented with aunique set of questions, depending on the audience being targeted.Questions are specific to the selected audience. A user may additionallywrite his/her own questions or select the default questions alreadystored in the database. For example, when designing evaluations for theResident audience (residents evaluating residents (peer-to-peeraudience)) the user may be presented with a general list ofcompetencies. Each competency will have its own set of questions 183.When designing evaluations for the Attending audience (attendingsevaluating residents) the user may be presented with two sets ofcompetencies: a set of Core Competencies 185 with six uniquesub-categories of competencies; and, a set of Secondary Competencies 186with multiple sub-categories of competencies. To user two-tieredquestions (as used in standard ABIM resident evaluations) the user firstselects the “0-9 Sliding Scale” listed in FIG. 8( b)(3). Two-tieredquestions are designed with a negative statement on the left and apositive statement on the right. All these questions use a 9-pointsliding scale as defined herein.

The fourth step of designing an evaluation involves the step of viewingand confirming the evaluation as it will appear when published (i.e.,available to residents and attendings) and, is the last step beforecompleting the customized evaluation. The user will then by presentedwith a ‘Finish’ option (not shown) to confirm the evaluation, or a‘Back’ option to return to Step 3, or ‘Cancel’ to exit and return to theEvaluations menu.

Even though selecting ‘Finish’ ends the evaluations design stage, it isunderstood that changes may still be made to the evaluation. Once the‘Finish’ option is selected, the custom evaluation is placed into astorage Evaluation Library 18 d in evaluations database 18 c (as shownin FIG. 1). All stored evaluations may be subsequently accessed by theuser by selecting the ‘Edit Evaluations’ menu option as will bedescribed in greater detail herein. It is noted that a ‘published’evaluation cannot be edited, however it can deleted and/or retired fromcirculation. In order to make an evaluation available to residents andattendings, the user must “Publish” the evaluation. To publish anevaluation, the user enters the ‘Edit Evaluations’ menu as will beexplained with respect to the interface 190 shown in FIG. 8( d). Fromthis interface, the user may select the ‘Publish’ option associated withthe desired evaluation. The user will then be prompted with aconfirmation screen (not shown) in order to confirm readiness to publishthe evaluation.

Edit Evaluation

Returning to the Evaluations Build interface 110 of FIG. 7, the user mayselect an option 114 to edit evaluations. In response to selectingoption 114, the user is presented with a downloaded web-based interface190 For viewing, editing, or deleting an existing evaluation 199.

Particularly, in view of FIG. 8( d), there is presented the contents ofthe exiting library of evaluations enabling the user/administrator toview, edit, delete, and publish existing evaluations. A user may alsochange the status of a voluntary evaluation to non-voluntary. The EditEvaluations option is presented as a table 191 that includes a listingof the existing library of evaluations for the client to view, edit,delete, and publish. The headings of table 191 include the Name of theEvaluation 192, i.e., the name of the evaluation as it was typed by theadministrative user in the first evaluation design step. The name of anevaluation can be Edited until the evaluation is published. Once anevaluation is published the name cannot be changed; the Target Audience193, i.e., referring to the evaluator or the person completing theevaluation; the Status 194 indicating a particular evaluation as being“Published” and the respective date the evaluation was published, or astatus of “Not Published”. As mentioned, an evaluation must be publishedin order to make it available for general user. Once the evaluation hasbeen published, this link appears as

and the evaluation can no longer be edited. A user may still delete,view, or change voluntary status of a published evaluation; and, variousActions 195 such as view 196 a, edit 196 c, delete 196 e, publish 196 d,and, voluntary/non-voluntary 196 b, each action's availability 196 bbeing dependent upon the status of the evaluation. The ‘View’ link 196 ato the right of the desired evaluation is provided to enable display orprint of a template of how the evaluation will appear to the audience.The Edit link enables editing of an existing evaluation, e.g., bymodifying the elements of the evaluation before it has been published.Once the evaluation has been published, this link is no longer availableand appears as 196 c. The deletion of an exiting evaluation is performedby selecting the ‘Delete’ action 196 e to the right of the selectedevaluation. Once an evaluation is deleted, it can no longer be used forfuture evaluation assignments. The data from a deleted evaluation ispreserved and can always be retrieved in a Report as will be hereinafterdescribed in greater detail. Selection of the publish link 196 dpermanently publishes the evaluation for general user. Once theevaluation has been published, this link appears as “

” and the evaluation can no longer be edited. One may still delete,view, or change voluntary status of a published evaluation. TheVoluntary/Non-voluntary link 196 b is a toggle option allowing a user toselect the display of the evaluation to residents/attendings in order tocomplete a voluntary evaluation. The ‘Non-voluntary’ option will onlydisplay the evaluation when it is assigned to residents/attendings, forexample.

Assign Evaluation

Returning to FIG. 7, from the build evaluations interface 110 the usermay select an option 116 to assign evaluations completely on-line. Inresponse to selecting option 116, the user is presented with adownloaded web-based interface 200 such as shown in FIG. 9( a) providingevaluation assignment mode functionality 203 for scheduling theassignment of an evaluation to residents or attendings, for example.Preferably, in response to selection of the schedule evaluationassignments functionality 203, an interface 208 is presented such asshown in FIG. 9( b) which illustrates the types of assignmentevaluations to residents including: residents of attendings 202,peer-to-peer 204, e.g., resident of resident or attending of attending;residents of program 206; and, residents of medical students 209.Referring back to FIG. 9( a), the assignment scheduling process 203enables assignment of evaluations to medical doctors, residents, nursesand fellows is preferably accomplished in a manner to provide 360 degreefield of evaluations within the hospital environment, e.g., manuallyassigning evaluations to a group or to an individual, e.g., according toan individual's name, sub-specialty, or post-graduate year; assigningevaluations by a cycle, e.g., month, quarter, year, or a specific dateor date range; monitoring pending evaluations and compliance; providingautomatic e-mail notification of assignments; providing automatic e-mailreminders 207, e.g., of incomplete evaluations; and ability to provideusers log on ability to complete an ad-hoc evaluation.

In an example implementation, in response to selection of the schedulingevaluation assignment option 203, a user may select the audience, e.g.,a resident or attending. With respect to selection of ResidentAssignments, the user's browser receives a web-based communicationcomprising an interface 210 as shown in FIG. 10 for effectingassignments of evaluations to residents. Likewise, selection of theAttending Assignments option will initiate a web-based download of aninterface (not shown) for effecting assignment of evaluations toattendings. From the example interface 210 depicted in FIG. 10,functions are executed for specifying the assignment of an evaluation toResidents, for example, who will be evaluating attending(s).

In a first step, as shown in the interface of FIG. 10, a user firstselects the category of evaluations 213, i.e., selecting the type ofevaluation that will be assigned to the evaluator. This will be one offour major categories of evaluations as will be described: 1) 360°Evaluations; 2) Peer-to-Peer Evaluations; 3) Non-Peer; and 4) Program.

The 360° Evaluations type basically permits the assignment ofevaluations to all individuals involved in the global delivery ofpatient care and medical education. This will include the Resident,Attending, Fellow, Nurse, and medical Student. In addition, this modulewill facilitate the assignment of Self-Evaluations for cross-comparisonto peers, supervisors, and subordinates. Initially, the administratormust select the “Focus Audience” to determine the person(s) that isbeing evaluated by all other in the 360° circle. After the FocusAudience is selected, there is a multi-step process for assigning an360° evaluation. An example provided herein with respect to FIG. 10focuses on the Resident as the Focus Audience.

Specifically, in the first step 213, the Evaluation Period and RotationName is selected via drop down menu 215. Then, in a second step, theuser selects the date range to assign evaluations, e.g., either byselecting a range of dates based on the program's rotation blocks (RB)217, or, may manually specify a start and end date in entry fields 219.In addition, the user may enter in an entry field, the name of rotationunder consideration. In a third step 220, the user may select Residentswho will be the Focus Audience, i.e., who complete the evaluation.Preferably, the residents may be selected by name or by post-graduateyear (PGY). The selected individuals will be the focus of all assignedevaluations.

Specifically, in a view of FIG. 10, the “Available Residents” list 222includes the names all residents currently available throughMYEVALUATIONS.COM for the particular hospital/department. Names may beadded or deleted from by returning to the “Main Menu” and selecting theoption “Manage User Profiles” as described herein. It should be notedthat Residents who are included in the “Manage User Profiles” list, butwho have completed their PGY term, will not appear in this list.Further, an “Available PGYs” list 224 is displayed to include allpost-graduate years (PGYs) available to the Department (e.g. InternalMedicine will have PGY-1 through PGY-3). This option thus enables theselection of a group of residents from the desired PGY. To select thisoption, the user selects the radio button next to “Available PGYs” andselect as many PGYs as needed. As further shown in FIG. 10, Add 225 a,remove 225 b, Add All 225 c and Remove All 225 d selection blocks areprovided to enable the respective moving of the selected name(s) or PGYsfrom the Available box to the “Selected Residents” box 226, the move ofthe selection(s) from the “Selected Residents” box to the Available box,the addition of all the names or PGYs from the Available box to the“Selected Residents” box, or the deletion of all the names from theAvailable box to the “Selected Residents” box.

In a fourth step 230, the Attending(s) who is/are subject of theevaluation are selected by name or by specialty. Thus, an “AvailableAttendings” list 232 may be displayed which provides the names allattendings currently available through MYEVALUATIONS.COM. Addition ordeletion of names from this list may be performed via the “Manage UserProfiles” option of the “Main Menu” (FIG. 4). This option may be used toselect a group of attendings according to name. A further “AvailableSpecialties” list 234 is also presented which provides the names allspecialties currently available to the department. This option may beused to select a group of attending from the desired Specialty bymarking a radio button next to “Available Specialties” and selecting asmany specialties as needed. As in the selection of Available Residents,Add 235 a, remove 235 b, Add All 235 c and Remove All 235 d functionblocks are provided to enable the respective selection or removal of theselected Attending(s)/specialties name(s) to the “Selected Attendings”box 236.

In a fifth step 240, the option is provided for enabling an evaluationto be assigned in the other direction, e.g., attendings to evaluateresidents. This may be accomplished in the manner specified inaccordance with the functions provided in steps 3 and 4. However, inthis instance, attendings who will be the evaluators are selected, andresidents to be evaluated, are selected. The particular evaluation maybe selected via the drop down menu 242 in FIG. 10.

It is understood that Nurses may complete this evaluation and may beselected by name or by specialty. The selected individuals will completeevaluations on resident/s specified in the second step of FIG. 10. An“Available Nurses” list (not shown) may be displayed which provides thenames all the nurses currently available through MYEVALUATIONS.COM withaddition/deletion of names provided via the “Manage User Profiles”option from the main menu. Additionally, the same features of step threefor Attendings may be applied to select the Fellow/s by name or byspecialty. The selected individuals will complete evaluations onresident/s specified in the second step. Similarly, the same features ofstep three for Attendings may be applied to select the Medical studentswho will complete this evaluation on resident/s specified in step two,for example.

The Peer-to-Peer category focuses on assigning evaluations from thetarget audience to others in the same target audience group, forexample, residents evaluating other residents. This comprises a 5-stepprocess for assigning an evaluation to an individual or group ofresident(s). A first step comprises selecting the evaluation to be usedby the resident(s) to evaluate other resident(s). In a second step, thedate range to assign evaluations is selected, as step 2 of FIG. 10. Thatis, a range of dates based on the program's rotation blocks (RB) may beselected, or manual start and end dates may be specified. In a thirdstep the resident(s) who will be assigned the evaluation are selected byname or by post-graduate year (PGY). The selected individuals is takenfrom the “Available Residents” list which has the names all residentscurrently available through MYEVALUATIONS.COM. It is understood thatResidents who are included in the “Manage User Profiles” list, but whohave completed their PGY term, may not appear in this list. An“Available PGYs” list is provided to include all post-graduate years(PGYs) available to the Department (e.g. Internal Medicine will havePGY-1 through PGY-3) and selection of a group of residents from thedesired PGY may be alternately performed. In a fourth step, theResident(s) to be evaluated are selected by name or by post-graduateyear (PGY). The selected individuals will be evaluated by theresident(s) specified in the third step.

The Non-Peer category focuses on including any target audienceevaluating another (different) target audience. For example, Residentsevaluating Attendings. This also may comprise a 5-step process forassigning an evaluation to an individual or group of resident(s). Afirst step involves Selecting a Resident Evaluation to be used by theresident(s) to evaluate the attending(s). A step two involves selectingan evaluation period, e.g., a date range to assign evaluations which maycomprise a range of dates based on the program's rotation blocks (RB),or specify manual start and end dates. A third step involves selectingthose Residents, for example, who will complete this evaluation, e.g.,selected according to name or by post-graduate year (PGY). The selectedindividuals will be assigned the evaluation and may be selected from the“Available Residents” list or the “Available PGYs” list. The next stepinvolves selecting those attending(s) who are to be evaluated, e.g.,either by name or by specialty. The selected individuals will beevaluated by the resident(s) specified in Step three either through the“Available Attendings” list or the “Available Specialties” list asdescribed herein. An optional timesaving step for assigning the aboveResident(s) for evaluation by the above Attending(s) using a specifiedevaluation may additionally be preformed, if selected, or otherwise,only the Resident(s) will be evaluating Attending(s).

The Program category focuses on the assignment of the target audience toevaluations unrelated to another target audience, e.g., residentsevaluating the cafeteria. Thus, in a first step, the evaluation to beused by the resident(s) to evaluate the Program is selected. Then, thedate range to assign evaluations based on the program's rotation blocks(RB), or manual start and end dates is specified. Thirdly, the Residentswho will complete this evaluation are selected in the manner asdescribed herein (e.g., by name or by post-graduate year (PGY)).

In sum, from the exemplary interface 210 of FIG. 10, once the categoryhas been selected and the various steps followed, the administrator maythen press Submit 245 in order to schedule and assignment. By pressingthe Submit button located at the bottom of each submission page (FIG.10), the system instantly and automatically send an e-mail notificationof a new assignment to each person selected in the target audiencefield.

Referring back to FIG. 9( a), in a further example implementation,selection of the Manage Evaluation Assignments option 205 from theEvaluation Assignments menu 200 provided in FIG. 9( a) will result inthe web-based download of an interface 250 shown in FIG. 11 for enablingthe Management of previously assigned evaluations. Specifically, asshown in FIG. 11, functionality is provided from this screen, to enableusers to sort through the list of assignments and delete previousassignments, or, to enable users to delete an individual or a group ofassignments. As shown in FIG. 11, an evaluation assignment may beselected by the name of assignee, date assigned, or by the name of theperson having made the assignment. The administrator is presented with atable of assignments 251 listing all evaluation assignments 252 sortedfrom top to bottom by order of submission date. A user/administrator mayselect a name from a “Look-up by name” option 254 in order to find thename of a current user and display all evaluations assigned to andpending on the selected person. Otherwise, the “Select an Administrator”option 255 may be used in order to find all evaluations assigned by aspecific administrator. In all instances the administrator is given theoption to Edit or Delete a specific assignment from the table. Deletingan assignment will completely remove it from the database of assignmentspending.

Returning to the Evaluations Build interface 100 of FIG. 7, the user mayselect an option 118 to manage the questions database 18 a of FIG. 1which initiates the downloading of a Manage Evaluations Questionsinterface 260 as shown in FIG. 12. Specifically, FIG. 12 illustrates themechanism for Adding 261, Deleting 263 or retiring 266 question to andfrom the library of on-line evaluation questions. Questions areidentified by question Ids 269 in a table 268 and categorized bycompetencies for each target audience and category in the medicalprofession. For instance, as shown in FIG. 12, the user will firstselect a category from the pull-down menu 262 of questions categories,i.e. competencies. The competencies include, but are not limited to:Core Competencies including those relating to Patient Care; MedicalKnowledge; Interpersonal and Communication Skills; Professionalism;Practice-Based Learning And Improvement; and, Systems-Based Practice;and, secondary Competencies including those relating to: Availability;Clinical Judgment; Clinical Skills; Enthusiasm and Responsiveness;Humanistic Qualities; Medical Care; Moral and Ethical Behavior;Personality; Responsibility; and, Teaching Skills. It should beunderstood that the question types may be directed to any of the targetaudiences including Residents, Attendings, Medical Students, Fellows,and Nurses covering any of the evaluation categories, e.g.,Peer-to-Peer, Non-Peer, Program, etc. In the preferred embodiment, eachnew department/client is setup with a default set of evaluationquestions for their question database. As further shown in FIG. 12, theuser may select the Add Question option 265 which initiates download ofa web-based display an example display 270 which is shown in FIG. 13including entry fields for enabling the addition of one or morequestions 272 and associated question IDs 269 pertaining to a selectedquestion category 275, i.e., core or secondary competency.

Report Generation

From the main menu interface 60,70 of FIGS. 3( a) and 3(b), auser/administrator may initiate the complete on-line/web-basedgeneration of a report based on the data collected via the onlineevaluations by selecting the Reports option 65 which initiatespresentation of a Reports Selection interface 300 such as shown in FIG.14. As part of this functionality, implemented by the reports generationmodule 27 of FIG. 1, a user may generate clear, concise, andcomprehensive on-line reports from the data collected and, additionally,facilitate tracking of medical doctor, resident, nurse and fellowcompletion of evaluations. In collecting data from evaluations, theon-line reporting feature is an important and indispensable feature.Each group evaluated will have a unique set of evaluation reports.

From the interface 300 of FIG. 14 there are provided report lists 304 a,304 b for residents and attendings, respectively. All reports forResidents, Medical Students, Fellows and Nurses will have the sameformat. In describing each report, the appropriate group is substitutedin each description. Some of the reports that may be generated on-linefor residents include: 1) Individual Resident Evaluations 310; 2)Summary of Group/Resident Evaluations 320; 3) Summary Peer-to-PeerEvaluations 330; 4) Summary Program Evaluations 340; 5) Trending ofResident Performance 350; 6) Summary of Resident's Core Competencies360; 7) Overdue Evaluations 370; 8) Completion Status 380; 9) Class Rank390; and 10) Early Warning Reports 400 and Resident Comments 410.

When selected, the Individual Resident Evaluations report 310 initiatesfunctionality for generating a detailed report on an individual'sevaluation performance. An example Individual Resident Evaluationsreport 312 is shown in FIG. 15( b) wherein each evaluation is listedseparately, including a table 313 providing the questions, scores andcomments. Each score is compared to the person's peers in the same group(Group score, e.g., PGY or Specialty) and also to all (Total score).When attendings are the evaluator there will be an additional columnlisting the Attending's average score for the specific and the selecteddate range. The last page of the report may include a statisticalanalysis summarizing the “n” value, mean, median, variance, height andlow (not shown). To provide this functionality, a web-basedcommunication such as the Individual Resident Evaluations reportinterface 314 shown in FIG. 15( a) is provided. As shown in FIG. 15( a),for generating this report, there are four selection criteriaincluding: 1) entry fields 311 for the entry of a start and end daterange; 2) pull-down menu 316 for the selection of a Resident, e.g., bychoosing a name from the list of all residents; 3) pull-down menu 317for the selection of an Evaluation, e.g., by choosing one evaluationfrom the list of all evaluations; and 4) selecting the type ofEvaluation including a radio button option 318 a, for selecting “AllResults” (default) to include all the reported data or the selectedresident including results from evaluations completed voluntarily andthose assigned by the Program, for example; and a radio button option318 b for selecting an “Assigned Results” to include only data fromassigned evaluations. This option excludes results from evaluationscompleted voluntarily.

When selected from FIG. 14, the Summary of Group/Resident evaluationsreport 320 initiates functionality for generating a detailed summaryattending evaluation report on an individual or group of selectedresidents. An example Summary of Group/Resident Evaluations report 321is shown in FIG. 16( b) wherein each evaluation is listed separately,including a table 322 which includes summary data on each selectedresident, in reference to each evaluation question. The results arecompared to the residents' peers in the same PGY (Group score) and toall residents (Total score) including a last page (not shown) whichincludes the statistical analysis, summarizing the n value, mean,median, variance, high and low values. To provide this functionality, aweb-based communication such as the Summary of Group/Residentevaluations report interface 323 shown in FIG. 16( a) is provided. Asshown in FIG. 16( a), for generating this report, there are fourselection criteria including: 1) entry fields 234 enabling the entry ofa start and end date range; 2) a mechanism 325 for selecting residentsto review including Add, Add All, Remove, Remove All functionality forchoosing residents as described herein, particularly by selection of aname, for example, from the list of Available Residents or selectingresidents based on PGY level, by pointing and clicking on a desiredname(s) or PGYs and then selecting “Add>>” button to select the name(s);or, selecting all residents names from the list of Available Residentsby clicking the “Add All>>” to select all the names. Furtherfunctionality for removing some or all of the selected residents namesis provided; 3) a pull-down menu 327 for enabling the selection of anEvaluation, e.g., by choosing one evaluation from the list of allevaluations; and 4) selecting the type of Evaluation including a radiobutton option 328 a, for selecting “All Results” (default) to includeall the reported data for the selected resident including results fromevaluations completed voluntarily and those assigned by the Program, forexample; and a radio button option 328 b for selecting an “AssignedResults” to include on data from assigned evaluations. This optionexcludes results from evaluations completed voluntarily.

When selected from FIG. 14, the Summary Peer-to-Peer Evaluations report330 initiates functionality for generating a detailed summary attendingevaluation report on an individual or selected residents (e.g., by PGY).An example Summary Peer-to-Peer Evaluations report 331 is shown in FIG.17( b) which includes summary data 332 on each selected resident, inreference to one or more evaluation questions. The results are comparedto the residents'peers in the same PGY (Group score) and to allresidents (Total score). Also included is a summary of anonymouscomments written by peers on the selected residents. Comments arereported anonymously without reference to author, date, time orrotation. The last page of the report includes a statistical analysissummarizing the “n” value, mean, median, variance, high and low values.To provide this functionality, a web-based communication such as theSummary Peer-to-Peer Evaluations report interface 333 shown in FIG. 17(a) is provided. As shown in FIG. 17 (a), for generating this report,there are four selection criteria including: 1) entry fields 334enabling the entry of a start and end date range; 2) a mechanism 335 forselecting residents to review including Add, Add All, Remove, Remove Allfunctionality for choosing residents as described herein, particularlyby selection of a name, for example, from the list of AvailableResidents or selecting residents based on PGY level, by pointing andclicking on a desired name(s) or PGYs and then selecting “Add>>” buttonto select the name(s); or, select all residents names from the list ofAvailable Residents by clicking the “Add All>>” to select all the names.Further functionality for removing some or all of the selected residentsnames is provided; 3) a pull-down menu 336 for enabling the selection ofan Evaluation, e.g., by choosing one evaluation from the list of allevaluations; and 4) selecting the type of Evaluation including a radiobutton option 338 a, for selecting “All Results” (default) to includeall the reported data for the selected resident including results fromevaluations completed voluntarily and those assigned by the Program, forexample; and a radio button option 338 b for selecting an “AssignedResults” to include on data from assigned evaluations. This optionexcludes results from evaluations completed voluntarily.

When selected from FIG. 14, the Summary Program Evaluations report 340initiates functionality for generating a detailed Summary Programevaluation report from the perspective of selected residents (e.g., byPGY). An example Summary Program evaluation report 341 is shown in FIG.18( b) which includes summary data 342 on selected PGY(s), in referenceto each evaluation question. Also included is a summary of anonymouscomments written by each resident. To provide this functionality, aweb-based communication such as the Summary Program evaluation reportinterface 343 shown in FIG. 18( a) is provided. As shown in FIG. 18( a),for generating this report, there are four selection criteriaincluding: 1) entry fields 344 enabling the entry of a start and enddate range; 2) a mechanism 345 for selecting a PGY by choosing “AllPGYs”to include all residents; otherwise, choosing a specific PGY inorder to restrict the search to one group (PGY); 3) a pull-down menu 346for enabling the selection of an Evaluation, e.g., by choosing oneevaluation from the list of all evaluations; and 4) selecting the typeof Evaluation including a radio button option 347, for selecting “AllResults” (default) to include all the reported data for the selectedresident including results from evaluations completed voluntarily andthose assigned by Program, for example; and a radio button option 348for selecting an “Assigned Results” to include on data from assignedevaluations. This option excludes results from evaluations completedvoluntarily.

When selected from FIG. 14, the Trending of Resident Performance report350 initiates functionality for generating a detailed trending report onan individual resident's performance over the course of time (e.g., 6months to 5 years). Trending is limited to a single resident and asingle evaluation. An example Trending of Resident Performance report351 is shown in FIG. 19( b) which charts an individual resident'sperformance 352 over the course of time. To provide this functionality,a web-based communication such as the Trending of Resident Performanceevaluation report interface 353 shown in FIG. 19( a) is provided. Asshown in FIG. 19( a), for generating this report, there are fourselection criteria including: 1) a pull-down menu 354 for enabling theselection of a Resident chosen from a list of all residents; 2) apull-down menu 355 for enabling the selection, of a Evaluation chosenfrom a list of all evaluations; 3) a pull-down menu 356 for enabling theselection of a trend chosen from time periods such as 6-month, or 1-yearor up to 5-years; and 4) selecting the type of Evaluation including theradio button options 357, 358 such as described herein with respect toFIG. 17( a).

When selected from FIG. 14, the Summary of Resident's Core Competenciesreport 360 initiates functionality for generating a detailed summaryreport of an individual resident's performance in Core and Secondarycompetencies. To provide this report, a web-based communication such asthe Summary of Resident's Core Competencies evaluation report interface363 shown in FIG. 20 is provided. As shown in FIG. 20, for generatingthis report, there are four selection criteria including: 1) entryfields 364 enabling the entry of a start and end date range; 2) apull-down menu 365 for enabling the selection of a Resident chosen froma list of all residents; 3) a pull-down menu 366 for enabling theselection of a Competency from a drop-down list 367 of competencies.This may include selection of “All Competencies” to generate data on allcompetencies reported on the selected resident including the percentperformance in each competency followed by data comparisons to the Group(other residents in the same PGY) and the Total (all residents in theProgram), or selection of an individual competency to generate detaileddata on the selected competency, including a comparison to the Group andTotal; and, 4) selecting the type of Evaluation including a radio buttonoption 368 a, for selection “All Results” (default) to include all thereported data for the selected resident including results fromevaluations completed voluntarily and those assigned by the Program, forexample; and a radio button option 368 b for selecting an “AssignedResults” to include on data from assigned evaluations. This optionexcludes results from evaluations completed voluntarily.

When selected from FIG. 14, the Overdue Evaluations report 370 initiatesfunctionality for generating a detailed report of all evaluationsoverdue. In the preferred embodiment, as shown in FIG. 21( a), thisreport 372 is automatically generated and has no selection criteria.Particularly, the overdue evaluations reports provides an alphabeticallist of each resident 373, their pager number 374, the names 375 andperiod 376 of each overdue evaluation 377, and the total number of daysoverdue 378. This report is specifically utilized to enable contact ofeach resident directly (e.g., by pager) in order to encourage thecompletion of overdue evaluations. FIG. 21( b) illustrates an examplee-mail communication 59 sent to an evaluator assigned an evaluation 379currently due or overdue, depending upon its status. This statusdetermination and automatic e-mail generation may be periodicallyperformed in accordance with the executing overdue evaluation scanmodule 28 in FIG. 1.

When selected from FIG. 14, the Completion Status report 380 initiatesfunctionality for generating a detailed summary report 381, such asshown in FIG. 22( b), providing data 382 indication the total number ofevaluations assigned, completed and pending for each resident, and forenabling a review of the Completion Status of an individual resident orall residents in the program. To provide this functionality, a web-basedcommunication such as the Completion Status report interface 383 shownin FIG. 22( a) is provided. As shown in FIG. 22( a), for generating thisreport, there are four selection criteria including: 1) entry fields 384enabling the entry of a start and end date range; 2) a mechanism 385 forselecting residents to review including Add, Add All, Remove, Remove Allfunctionality for choosing residents as described herein, particularlyby selection of a name, for example, from the list of AvailableResidents or selecting residents based on PGY level, by pointing andclicking on a desired name(s) or PGYs and then selecting “Add>>” buttonto select the name(s); or, selecting all residents names from the listof Available Residents by clicking the “Add All>>” to select all thenames. Further functionality for removing some or all of the selectedresidents names is provided; and, 3) a pull-down menu 386 for enablingthe selection of an Evaluation, e.g., by choosing one evaluation fromthe list of all evaluations.

When selected from FIG. 14, the Class Rank report 390 initiatesfunctionality for generating a detailed summary report 391, such asshown in FIG. 23( b), providing data 392 pertaining to the ranking ofresidents by percent performance in a selected competency. Furtheroptions for enabling the choice to review ranking of a single competencyacross “All Evaluations”, or to review a cross-comparative report ofmultiple competencies within a single evaluation, is provided. Across-comparative report is a ranking of residents by a singlecompetency within a single evaluation. This ranking is further enhancedby comparing data from other competencies, and listing it next to theranking data. This facilitates the cross-comparison of performance inone competency with respect to other competencies (within the sameevaluation). To provide this functionality; a web-based communicationsuch as the Class Rank report interface 393 shown in FIG. 23( a) isprovided. As shown in FIG. 23( a), for generating this report, there arefour selection criteria including: 1) entry fields 394 enabling theentry of a start and end date range; 2) a mechanism 395 for selectingavailable PGYs by enabling choice of a specific PGY in order to includeall residents from a given class; 3) a pull-down menu 396 for enablingthe selection of an Evaluation, e.g., by choosing an individualevaluation from the list of all evaluations, or choosing “AllEvaluations”. This option to enables review of ranking across allevaluation types, or review a cross-comparative report within a singleevaluation. That is, to generate a cross-comparative report, the usermust choose a specific evaluation in this step so that data will beautomatically updated in the next step 4) to include competenciesrelated to the specific evaluation; 4) a pull-down menu 397 for enablingthe selection of a competency. Preferably, the competencies listed inthis pull-down menu 397 will depend on the evaluation selection typefrom the previous step. That is, as maintained in the system databases(FIG. 1), competencies are associated with a particular evaluationtype/audience. For example, when “All Evaluations” is selected in theprior step 3), the user is presented with a list of competenciesincluded in the chosen evaluation. When a specific evaluation isselected in the prior step 3) the user is presented with a list ofcompetencies included in the chosen evaluation. A single competency ischosen for ranking residents. In the report, the first ranking columnlists the chosen competency, followed by the percent performance in eachsubsequent competency (included in the chosen evaluation). Finally, instep 5), the type of Evaluation is selected from among the “All Results”(default) or “Assigned Results” options 398 as described herein.

When selected from FIG. 14, the Early Warning report 400 initiatesfunctionality for generating a detailed summary report 401, as shown inFIG. 24( b), of all residents' performance (e.g., score 402) withrespect to a minimum standard. For example, performance may be measuredacross all evaluations and may include “All Competencies” or a singlecompetency. This report may be advantageously used to quickly uncoverresidents who may be performing below a minimum standard, and may needinterventional support. To provide this functionality, a web-basedcommunication such as the Early Warning report interface 403 shown inFIG. 24( a) is provided. As shown in FIG. 24( a), for generating thisreport, there are four selection criteria including: 1) entry fields 404enabling the selection of a specific PGY in order to restrict the searchto one group or, “All PGYs” in order to include all residents; 2) apull-down menu 406 for enabling the selection of a competency for basinga minimum performance standard. Preferably, there are two options: an“All Competencies” option for generating data on all competenciesreported on the selected resident with the results reported in a summaryformat; or, a specific competency selected from all availablecompetencies, wherein the results are cumulative and only in referenceto the selected evaluation; 3) a pull-down menu 407 for selecting aMinimum Standard for evaluating the residents' performance. The minimumstandard is a percent value. Any resident performing less than theminimum will be included in the Early Warning Report; and, 4) theselection options 408 for the type of Evaluation including the “AllResults” (default) option or “Assigned Results” option as describedherein.

When selected from FIG. 14, the Resident Comments report 410 initiatesfunctionality for generating a detailed summary report 411, as shown inFIG. 25( b), of comments 412 written by attending physicians on selectedresidents, for example. Preferably, each attending comment includes thename and specialty of the authoring attending physician. To provide thisfunctionality, a web-based communication such as the Resident Commentsreport interface 413 shown in FIG. 25( a) is provided. As shown in FIG.25( a), for generating this report, there are four selection criteriaincluding: 1) entry fields 414 enabling the selection of a start and enddate range; 2) a mechanism 416 for selecting residents to reviewincluding Add, Add All, Remove, Remove All functionality for choosingresidents as described herein, particularly by selection of a name, forexample, from the list of Available Residents or selecting residentsbased on PGY level by pointing and clicking on a desired name(s) orPGYs; 3) the pull-down menu 417 for selecting an evaluation, e.g., bychoosing either the “All Evaluations” or an individual evaluation. Thisoption is used to review the comments written on a specific evaluation,or on all evaluations; and 4) buttons options 418 enabling the selectionof the type of Evaluation from among the “All Results” (default) optionor “Assigned Results” option as described herein.

It should be understood that the report generation module 27 (FIG. 1)provided herein additionally is used to generate the following AttendingEvaluation reports selectable from the menu list 304 b of FIG. 14including: Individual Attending Evaluations 420; Summary of AttendingEvaluations 425; Summary Peer-to-Peer Evaluations 430; Summary ProgramEvaluations 435; Trending of Attending Performance 440, Summary ofAttending's core Competencies 445; Overdue Evaluations 450; CompletionStatus 455; Class Rank 460; and, Attending Comments 465. It isunderstood that these Attending reports are generated in a mannerlargely identical to the manner in which their counterpart ResidentEvaluation reports are generated.

Procedure Tracking

Referring back to FIG. 3( b), there is provided a “Procedures” option64, the selection of which initiates download of a web-basedcommunication providing a Procedures Menu Interface such as shown in theexample Procedures Menu Interface 500 of FIG. 26. As shown in FIG. 26,the web-based interface 500 provides each user with a sub-menu ofoptions that facilitates the on-line collection of all data pertainingto procedures completed by residents and medical students. Thisfunctionality is performed by the procedure build and tracking executionthreads 25 as shown in FIG. 1. For a user/administrator, these selectionoptions include an option 502 for submitting a new or completedprocedure on-line; an option 504 for modifying submitted procedures; anoption 506 for modifying the database of available procedure types andenabling design of a procedure; and, an option 508 for selecting asupervisor(s), i.e., identifying attendings qualified as supervisors.

Submit New Procedures

The person completing the procedure logs in to MYEVALUATIONS.COM® andselects a procedure to mark as completed. When selected from theProcedures Menu Interface 500 of FIG. 26, the option 502 for submittingnew procedures initiates downloading of a web-based communicationproviding the submit new procedure interface 510 such as the exampleinterface provided in FIG. 27. It is from this interface 510 of FIG. 27that enables a user to add a new completed procedure on-line. Based onthe Attending identified as qualified to certify procedures, the systempresents the user with the names of persons qualified to certify thecompleted procedure. The user then completes all the details of theprocedure by entering the information as specified in the entry fieldsprovided via the interface 510. Then the user submits the completed formvia a secure Internet connection for storage in the procedures database34. Once submitted, the person selected as the certifying individualwill receive a message in the MYEVALUATIONS.COM® mailbox, which messageprovides the certifier with the option to Accept or Reject the procedurecompletion form. If Accepted, the user receives credit for completingthe procedure. More particularly, with reference to FIG. 27, the userenters the following information: a Name field 512 for entering the nameof the person performing the procedure. For users such as residents andmedical students, the name field will display the First name and Lastname. It is a fixed field based on the user login information. For userssuch as administrators, the name field comprises a drop-down menu optionincluding the names of all residents and medical students with the namedisplayed as Last name, First name; a Select procedure field 514 whichprovides a pull-down menu 515 displaying procedure names available forcertification. This fluid will only display procedures marked as“Included” in the procedure database. For a resident, after logging in,only procedures marked as “Yes” in an Inc-Residents field (not shown)will be displayed. When a medical student logs-in, only proceduresmarked as “Yes” in the Inc-Med_Stu field will be displayed. TheCompleted field 514 is provided to display the total number of completedprocedures 514 a, i.e., a count for the selected individual withreference to the selected procedure. A pending field 514 b displays anumber result of pending procedures, i.e., the number of requiredprocedures minus the number of Completed procedures((Required)-(Completed); If the number of completed is greater than thenumber required, then this value will be set to zero. An option 516 isadditionally provided via interface 510 that provides details about howto perform the selected procedure. The “Supervised by” field 517displays the names of the Supervisors. Persons qualified to besupervisors include: Certified Residents and Attendings selected assupervisors. Certified Residents are residents who complete the numberof procedures required to be certified and the Procedures Database willinclude a certification number for those certified residents maintainedin the “Supervised by” field. For Attendings, a separate module forAttendings may be added or removed from this list; see “SelectSupervisors”. Further included is a Patient name entry field 519 whichreceives a plurality of character spaces for free-form entry of thepatients name. An MR# text entry field 520 is additionally provided forreceiving alpha-numeric characters for free-form entry of the patientsmedical record number. The date completed field 524 is additionallyprovided which comprise drop-down fields for selecting the month, dayand year, respectively. As further shown in the submit new procedureinterface 510 of FIG. 27, individual text fields 525 a-525 c areprovided for free-form entry of a patient's diagnosis. Furthermore,individual text fields 525 d-525 f fields are provided for free-formentry of indications to do the procedure. Furthermore, Yes/No radiobuttons 527 are provided for verifying whether consent was required todo the procedure. Further, a Complications Note field 528 is providedfor free-form entry of any complication associated with the procedure. AComments/Notes field 529 are additionally provided for free-form entryof any comments. After entry of each of the required fields, the usermay then Submit the procedure information by selecting a displayedsubmit icon. This will function similarly to an evaluation submissionand is stored in the procedures database 34 of FIG. 1. The SupervisingResident and Attending will receive an assignment in the sectionEvaluation/Procedures to be Acknowledged. The supervisor then has theoption to modify or, to Approve or Reject the completion of theprocedure. Once Approved, the resident or medical student will receivecredit for the procedure as completed. It should be understood thatevaluation questions are optionally included. From the “SelectProcedure-Eval Questions” menu option the administrator may selectstandard evaluation questions that will be included in the proceduresurvey. It should be understood that users will have access to thefollowing other Procedure Sub-Menu options: including an option (notshown) for viewing a detailed history of procedures; and, and option forviewing a Summary of completed reports.

When selected from FIG. 26, the option 504 for modifying submittedprocedures initiates functionality for generating a detailed summary, asshown in the example web-based communication 530 such as shown in FIG.28, providing a list of selected medical students/residents submitters531, their associated submitted procedures in column 532, and the statusof the procedures 533. Any action that is required for the procedure maybe edited by selecting an edit action 534 next to the selected procedurewhich returns the user/administrator to the submit new procedure screen(FIG. 27) for modifying/editing the data entered.

It should be understood that functionality is provided for designing aprocedure. That is, when selected from FIG. 26, the option 506 formodifying submitted procedures initiates functionality for generating adetailed list of all procedures 552, as shown in the example web-basedcommunication 550 such as shown in FIG. 29. For each procedure, certaincriteria may be modified. For example, by user selection of the optionrequired field 556 a will initiate toggling of that particular procedureas being required/non-required. Likewise, user selection of the optionresident field 556 b will initiate toggling of that particular procedureas being required for a resident or not. Similarly, user selection ofthe option medical student field 556 c will initiate toggling of thatparticular procedure as being required for a medical student or not.Further action 557 may be taken to edit that particular procedure, forexample, changing the procedure's name, changing the minimum number ofprocedures required to be performed for graduation or the certification,or by changing that procedures requirement as being available toresidents, medical students or, both. Further functionality 558 isprovided to delete that particular procedure from the list of procedures552.

Further with respect to FIG. 29, a button option 555 is provided fordesigning a procedure and adding it to the procedures database 34 ofFIG. 1. Selection of option 555 particularly initiates generation of aDesign a Procedure screen display 560 such as the example display shownin FIG. 30. From this display 560, an administrator may perform thefollowing functions: enter a new procedure's name in entry filed 561;entering the minimum number of procedures required to be performed forgraduation 562 or the certification 563, or radio button selectionoptions 565 specifying whether the new procedure is to be available toresidents, medical students or, both.

Referring back to FIG. 8, there is provided the selection option 508 forselecting a supervisor(s), i.e., identifying attendings qualified assupervisors. FIG. 31 illustrates the example web-based communicationproviding an interface 570 for selecting an attending who is tosupervise the procedure. As shown in FIG. 31 functionality 572 isprovided to enable selection of attendings to review includingmechanisms for Adding, Adding All, Removing, and Removing All availableattendings, as described herein, particularly by selection of a name,for example, from the list of Available Attendings and then selecting“Add>>” button to select the name(s); or, selecting all residents namesfrom the list of Available Residents by clicking the “Add All>>” toselect all the names. Further functionality for removing some or all ofthe selected attendings names is provided.

While the invention has been particularly shown and described withrespect to illustrative and preformed embodiments thereof, it will beunderstood by those skilled in the art that the foregoing and otherchanges in form and details may be made therein without departing fromthe spirit and scope of the invention which should be limited only bythe scope of the appended claims.

What is claimed is:
 1. A program storage device readable by machine, thestorage device comprising instructions executable by the machine toperform a method for evaluating the competency of an evaluatesassociated with an academic hospital in one or more competencycategories, the method comprising the steps of: storing a plurality ofevaluation questions within a first database; associating each questionwith one or more of the competency categories, each competency categoryassociated with a function of individuals within the academic hospital;generating an online evaluation comprising a plurality of questions forevaluating an evaluatee; automatically assigning the evaluation to anevaluator for completion online, assignment being according to a profileinformation of the evaluator and the evaluatee; tracking whether theevaluator has completed the evaluation; receiving one or morecorresponding responses from the evaluator; storing the responses in asecond database; retrieving one or more of the responses from the seconddatabase and generating a first report at least partially based thereon;generating a second report indicating, for at least one assignedevaluation, at least one of: a due date for completion, a remainingperiod until the due date, and an elapsed period since the due date;wherein the first report is an online accessible report comprising oneor more competency category scores generated based on the responses;wherein the evaluatee and evaluator comprise individuals associated withthe hospital; wherein the first database includes a user profileinformation for each individual associated with the hospital, the userprofile information including a length of time the individual has beenassociated with at least one of: the hospital, a department, anassignment, and another individual.
 2. The program storage device ofclaim 1, wherein the second database comprises responses from aplurality evaluations.
 3. The program storage device of claim 1, whereinthe first database comprises the second database.
 4. The program storagedevice of claim 1, further comprising the step of selecting at least oneevaluator and at least one evaluatee.
 5. The program storage device ofclaim 4, wherein the first database includes a user profile informationfor each individual associated with the hospital, the user profileinformation including at least one of: a name of the individual and alength of time the individual has been associated with at least one of:the hospital, a department, an assignment, and another individual; andwherein assigning the evaluation to the evaluator is based on the userprofile information of the evaluator and the evaluatee.
 6. The programstorage device of claim 1, wherein the evaluation questions includepre-generated questions and custom-generated questions. 7 The programstorage device of claim 1, wherein the competency categories comprisecore competencies and secondary competencies.
 8. The program storagedevice of claim 1, wherein each competency category comprises at leastone competency subcategory; and wherein the report comprises a scoreassigned to each competency sub-category based on the question responsesassociated with the competency sub category.
 9. The program storagedevice of claim 1, further comprising the step of: tracking whether theevaluation response associated with an assigned evaluation has beenreceived, and wherein the report includes an indication of receipt ornon-receipt of the evaluation response.
 10. The program storage deviceof claim 9, wherein each evaluation comprises a due date; and whereinthe indication of non-receipt includes: an indication of a number ofdays until the due date, or an indication of a number of days since thedue date.
 11. A computer implemented system for facilitating theevaluation of one or more individuals associated with an academichospital according to one or more competency categories, each competencycategory indicative of the individual's proficiency with respect to oneor more of: desired skills, attributes, abilities, subject matterknowledge and the completion of tasks, the system comprising: adatabase; a first interface in network connection with the database, thefirst interface for permitting an administrator to accomplish one ormore of the following: generating one or more questions for storage inthe database, associating one or more of the questions with one or morecompetency categories; selectively assigning one or more of thequestions to an evaluator for a purpose of evaluating at least one of:an individual and a group of individuals associated with the academichospital; a second interface in network connection with the database,the second interface permitting the evaluator to accomplish one or moreof the following: generating one or more responses corresponding to theassigned questions, storing the responses in the database according toat least one of: corresponding question, competency category, individualevaluated and group of individuals evaluated; wherein the firstinterface further permits the administrator to generate a reportindicating a level of proficiency of the individual or group ofindividuals in one or more competency categories.